A Constraint-Based Editor for Linguistic Scholars

Corpus linguistics is a branch of linguistics in which the scholars analyse large collections of electronic documents. For English there are many such corpora. The earliest and best known is the Brown Corpus [1]. The Linguistic Data Consortium at the University of Pennsylvania is collecting several hundred million words of English. The Text Encoding Initiative (TEI) [2] is specifying SGML Document Type Definition standards for corpus linguistics, which have already been put to use in the 100 million word British National Corpus (BNC) [3]. The International Corpus of English (ICE) [4], of which one of us (Meyer) leads the American effort, is collecting 1 million words of written and spoken English from each of 15 countries. To be useful for electronic analysis, documents once transcribed into electronic form must have some conventional markup inserted to mark the boundaries of whatever linguistic constructs are to be studied. Some of this markup, such as that implied by the natural language parsing, can be inserted by parsing programs with, perhaps, a small amount of human intervention to disambiguate the parse (cf. [5]). When the transcription is of spoken language, the state of natural language parsing is inadequate to this task, and even for written text some things do not yield to machine marking. For example, in order to support correct analyses, mis-spellings may be corrected by a (human) corpus editor, and the corpus must retain both the original and the ‘normalization’, both suitably marked. Therefore, perhaps after initial machine markup a document must have markup added by a human, typically with some linguistic training. This is a time consuming task, prone to logical errors if done without adequate software. Our program supports the addition of ICE markup in ways that make impossible the addition or deletion of markup in violation of constraints defined by the markup scheme, while at the same time permitting the full editing of the text itself. The constraints are deduced from a hierarchy we have placed on the ICE markup (none is officially specified), but are easily changed.